Representing de-duplicated file data

ABSTRACT

Providing a subset of de-duplicated data as output is disclosed. The subset of data includes a plurality of data segments. In some embodiments, the output includes segments of data stored across a plurality of containers in a de-duplicated storage system. Each container stores one or more data segments from the subset as well as one or more segments not from the subset. For each container that includes at least one data segment from the subset, the corresponding container data is included in the output in its entirety, including segments that are not from the subset. In the event that container data is included in the output, a value in a data structure that corresponds to the container is updated. The data structure includes a value for each container in the de-duplicated storage system that indicates whether the corresponding container data has been included in the output.

CROSS REFERENCE TO OTHER APPLICATIONS

This application is a continuation of co-pending U.S. patent applicationSer. No. 12/803,393, entitled REPRESENTING DE-DUPLICATED FILE DATA filedJun. 25, 2010 which is incorporated herein by reference for allpurposes.

BACKGROUND OF THE INVENTION

Deduplication storage systems such as described in U.S. Pat. No.6,928,526, entitled EFFICIENT DATA STORAGE SYSTEM, filed Dec. 20, 2002and issued Aug. 9, 2005, the disclosure of which is incorporated hereinby reference for all purposes, have been disclosed. In such systems, astream of data to be stored is divided into segments. Typically asegment is stored on the deduplication storage system only once, even ifthe segment occurs in more than one file or other object and/orotherwise occurs more than once in the data stream.

On occasion a need arises to generate an output stream comprising aspecified subset of a set of data that has been stored in adeduplication storage system. For example, a data owner may wish tocreate a tape (or other removable or non-removable media) archive of asubset of data stored on a deduplication storage system. One approachthat has been used to create a tape archive or other data streamcomprising such a subset stored in de-duplicated form, to conserve spaceon the destination tape or other media, is to “re-inflate” (i.e.,reverse deduplication and/or decompress) the data as stored on thededuplication storage system, feed the subset to a second deduplicationstorage system, then copy the subset as stored in de-duplicated form onthe second deduplication storage system to the tape or other media.However, this approach consumes processing resources and time (tore-inflate the data, for example) and the availability of a seconddeduplication system.

BRIEF DESCRIPTION OF THE DRAWINGS

Various embodiments of the invention are disclosed in the followingdetailed description and the accompanying drawings.

FIG. 1 is a block diagram illustrating an embodiment of a network andassociated data storage environment.

FIG. 2 is a block diagram illustrating an embodiment of a deduplicationstorage system.

FIG. 3 is a flow diagram illustrating an embodiment of a process forproviding as output a subset of data as stored on a deduplicationstorage system.

FIG. 4 is a diagram illustrating an example of an output stream asprovided in some embodiments.

FIG. 5 is a flow diagram illustrating an embodiment of a process forproviding as output a subset of data as stored on a deduplicationstorage system.

FIG. 6 is a flow diagram illustrating an embodiment of a process forreconstructing a subset of data using a copy of the subset as stored ona deduplication system.

FIG. 7 is a flow diagram illustrating an embodiment of a process forreconstructing a subset of data using a copy of the subset as stored ona deduplication system.

FIG. 8 is a flow diagram illustrating an embodiment of a process forreconstructing a file from a data stream comprising a copy of a subsetof data as stored on a deduplication system.

DETAILED DESCRIPTION

The invention can be implemented in numerous ways, including as aprocess; an apparatus; a system; a composition of matter; a computerprogram product embodied on a computer readable storage medium; and/or aprocessor, such as a processor configured to execute instructions storedon and/or provided by a memory coupled to the processor. In thisspecification, these implementations, or any other form that theinvention may take, may be referred to as techniques. In general, theorder of the steps of disclosed processes may be altered within thescope of the invention. Unless stated otherwise, a component such as aprocessor or a memory described as being configured to perform a taskmay be implemented as a general component that is temporarily configuredto perform the task at a given time or a specific component that ismanufactured to perform the task. As used herein, the term ‘processor’refers to one or more devices, circuits, and/or processing coresconfigured to process data, such as computer program instructions.

A detailed description of one or more embodiments of the invention isprovided below along with accompanying figures that illustrate theprinciples of the invention. The invention is described in connectionwith such embodiments, but the invention is not limited to anyembodiment. The scope of the invention is limited only by the claims andthe invention encompasses numerous alternatives, modifications andequivalents. Numerous specific details are set forth in the followingdescription in order to provide a thorough understanding of theinvention. These details are provided for the purpose of example and theinvention may be practiced according to the claims without some or allof these specific details. For the purpose of clarity, technicalmaterial that is known in the technical fields related to the inventionhas not been described in detail so that the invention is notunnecessarily obscured.

Providing as output, in a de-duplicated form, a specified subset of aset of data stored on the deduplication storage system, is disclosed. Insome embodiments, a segmented data stream received (or generated, forexample, by segmenting a received data stream) at the deduplicationstorage system is de-duplicated by storing fewer instances of eachunique segment than are received. Each segment instance is identified bya segment reference and stored in one of a plurality of containersidentified by a container identifier (“container ID”). An index ismaintained that maps each segment to the container in which it isstored, and for each file or other object included in the stored data anordered list of references to the segments comprising the file or otherobject. In some embodiments, if a subset of the data is desired to beprovided as output, in a de-duplicated form, the list of segmentreferences comprising each file or other object is included in theoutput. The index is then used to determine which container(s) includethe segment data associated with each file or other object in thesubset. As each container is identified, the corresponding containerdata for the entire container is included in the output stream and a bitmask or other data structure is updated to reflect that the container'sdata has been included in the output. As each subsequent file or otherobject is processed, the container(s) that included one or more segmentscomprising the file or other object are determined and the bit mask orother data structure is checked to determine whether the container(s)has/have been included already in the output. The container dataassociated with any container determined to contain a segment comprisinga file or other object in the subset that has been determined bychecking the bit mask or other data structure to not already have beenincluded in the output is written to the output stream and the bit maskor other data structure is updated. In this way, data segmentscomprising files or other objects included in the subset are provided asoutput, in de-duplicated form, quickly and efficiently, with minimaloverhead and without having to re-inflate the data.

FIG. 1 is a block diagram illustrating an embodiment of a network andassociated data storage environment. In the example shown, the networkenvironment 100 includes a plurality of client computer systems and/orother data sources represented in FIG. 1 by clients 102, 104, and 106.The clients are connected via a network 108 to a backup server 110configured to backup to other storage data stored initially on clientssuch as clients 102, 104, and 106 and/or other systems. While a backupserver 110 and associated clients 102, 104, and 106 are shown in FIG. 1,in various embodiments any source(s) of data to be stored may be used.In the example shown, the backup server 110 provides a backup datastream as output via a connection 112 to a deduplication storage 114.Deduplication storage 114 is configured in various embodiments toprovide as output, on a connection 116 to a destination media, system,and/or network connection 118 in the example shown, a specified subsetof data as stored in de-duplicated form on deduplication storage 114.For example, in some embodiment destination 118 comprises a tape orother removable storage media. In various embodiments, connection 112and/or connection 116 comprises network connections and/or othercommunication interfaces.

FIG. 2 is a block diagram illustrating an embodiment of a deduplicationstorage system. In the example shown, deduplication storage system 114includes a segment reference generation and duplication check engine202. For each segments received via connection 112 a (likely) uniquesegment fingerprint, for example a hash or other fingerprintrepresentative of the segment data, is generated. An index, list, orother set of segment fingerprints associated with previously storedsegments is checked. If the segment is a duplicate of a previouslystored segment a reference to the segment as stored previously isassociated with the file or other object with which the currentlyreceived segment is associated. Otherwise, the segment is added to asegment and index store 204 by storing the segment data in one of aplurality of containers, represented in FIG. 2 by containers 206 and208, and index data associating the container, for example by uniquecontainer ID, with the segment fingerprint of the segment data is storedin a metadata index 210. In some embodiments, the index data alsoincludes for each segment a segment ID or other data indicating alocation of the segment within the container in which it is stored. Ade-duplicated data subset generator 212 is configured to receive adesignation of a subset of data stored on deduplication storage 114 andprovide as output via connection 116 a data stream comprising thedesignated subset as stored in de-duplicated and, in some embodiments,compressed form on deduplication storage 114. In various embodiments,de-duplicated data subset generator 212 includes in the output datastream the container data stored in each container that includes one ormore segments comprising a file or other object included in the subset.In various embodiments, the container data is not decompressed orotherwise re-inflated prior to being provided as output, and instead iswritten to the output stream. In some embodiments, the data is providedas output in the form in which it is stored on deduplication storage114. In some embodiments, prior to being provided as output the data isprocessed into a form other than the form in which it is stored ondeduplication storage 114. In various embodiments, de-duplicated datasubset generator 212 includes in the output data stream one or more of afile (or other object) header for each file (or other object) includedin the subset; container data for any container(s) that include(s) oneor more segments comprising the file (or other object), if not alreadyincluded previously in the output stream by virtue of being associatedwith another file or other object in the subset; and one or more segmentreferences identifying the segment(s) comprising the file or otherobject. In various embodiments, one or both of segment fingerprintgeneration and duplication check engine 202 and de-duplicated datasubset generator 212 comprise one or more processes and/or modulesrunning on one or more processors comprising deduplication storagesystem 114. In various embodiments, segment and index store 204 compriseone or more storage disks, arrays of disks, and/or other data storagedevices and/or systems. In various embodiments, for each containeridentified as including one or more segments comprising a file or otherobject included in the subset, the de-duplicated data subset generator212 checks a bit mask or other data structure, stored for example in amemory or other storage location on deduplication storage system 114, todetermine whether the corresponding container data has been includedalready in the output stream. If so, only the file header and segmentreferences for the file or other object currently being processed arewritten to the output stream. If not, the container data is written tothe output stream and the bit mask or other data structure is updated toreflect the fact that the container data for that container has beenwritten to the output stream. In this way, the container data for acontainer that includes one or more segments required to be included inthe output is written to the output stream only once. Note that for eachcontainer the corresponding container data may, but will notnecessarily, include one or more segments not associated with any fileor other object included in the specified subset. In some embodiments,deduplication, garbage collection and/or other processes at adestination system to which the data included in the output stream isprovided is configured to remove from the data segments that are notassociated with a file or other object included in the specified subset.

FIG. 3 is a flow diagram illustrating an embodiment of a process forproviding as output a subset of data as stored on a deduplicationstorage system. At 302, an identification of the subset of data to beprovided as output in the de-duplicated form in which it is stored onthe deduplication storage system is received. For example, a directoryor subdirectory may be designated to be included in the output. At 304,a data stream that includes container data as stored on thededuplication storage system for each container in which one or moresegments comprising a file or other object included in the subset isprovided. In various embodiments, the output data stream includes foreach file or other object a file or object header; container data forany container that includes one or more segments comprising the file orother object, if not already included previously in the output stream;and one or more segment references identifying the segment(s) comprisingthe file or other object.

FIG. 4 is a diagram illustrating an example of an output stream asprovided in some embodiments. In the example shown, the output datastream 400 includes a file header 402 for a first file. Segmentscomprising the first file have been stored in this example in threecontainers, identified as containers 1, 2, and 3, respectively. As such,a container data 404 corresponding to container 1 is included in theoutput stream 400. Segment references 406 identify the segments incontainer 1 that are associated with the first file, in this example,the first, second, and third segments in the container. Likewise,container data 408 corresponding to container 2 is included in thestream 400, followed by segment references 410 to the first and thirdsegments in container 2, which in this example are associated with thefirst file. Next, container data 412 corresponding to container 3, andsegment reference 414 identifying the second segment in container 3 asbeing associated with the first file, are included in the stream 400.Later in the stream, a file header 416 identifying an n-th file in thesubset is included in the stream 400. The n-th file in this exampleincludes the first and second segments stored in the container 1. Invarious embodiments, the output stream generator by checking a bit maskor other data structure, e.g., as described above, would determine thatthe entire contents of container 1 had already been written to theoutput stream 400. As a result, the container data corresponding tocontainer 1 is not included again, and instead the segment references418 are included in the stream immediately following the file header416. Note the same would occur if the n-th (or any other) file includedone or more segments not referenced by any previously processed file (orother object) but that were included in a container the correspondingcontainer data of which had already been written to the stream 400, forexample by virtue of one or more other segments included in thatcontainer having been determined to be associated with a previouslyprocessed file (or other object).

FIG. 5 is a flow diagram illustrating an embodiment of a process forproviding as output a subset of data as stored on a deduplicationstorage system. In the example shown, starting with a first file to beincluded in the subset, a file header is written to the output stream(502). A next segment reference for the file is read (504). Thecontainer ID for the referenced segment is looked up (506). In someembodiments, both the container ID and a segment ID or other dataindicating a location of the referenced segment within the container arelooked up (506). It is determined whether a bit or other data valuecorresponding to the container ID is set, indicating that the associatedcontainer data has been written previously to the output stream (508).If not, the container data is read from disk (510) and written to theoutput stream (512), and the corresponding container ID bit or othervalue is set. In some embodiments, container data is written to theoutput stream in the same form in which it is stored on thededuplication storage system. If the container ID was already set in thebit mask (508), or once the container data has been read and written tothe data stream and the container ID set (510-512), the segmentreference read at (504) and the container ID looked up at (506) arewritten to the data stream (514). Successive segments comprising thefile are processed accordingly until a last segment comprising the filehas been processed (516). Once the last file in the subset has beenprocessed (518) the process of FIG. 5 ends.

FIG. 6 is a flow diagram illustrating an embodiment of a process forreconstructing a subset of data using a copy of the subset as stored ona deduplication system. In the example shown, the reconstruction isperformed on a destination deduplication storage system. In the exampleshown, a next header included in the data stream is read (602). If theend of the stream has been reached (604), the process ends. Otherwise,if the header is a file header (606) a “content stream” associated withthe file is opened (608) on the destination deduplication storagesystem. In some embodiments, the destination deduplication storagesystem is configured to use a content stream to store data comprisingthe file in de-duplicated form. If the header instead identifies thecurrent data in the stream as comprising container data (610), thecontainer data is written to disk on the destination deduplicationstorage system (612) and indexed. In some embodiments, a container ID(or a new one) may be assigned to identify the container uniquely on thedestination deduplication storage system. If the header comprises asegment reference (614), the reference is written (616) to the contentstream opened at (608). Finally, if the end of the current file has beenreached (618) the associated content stream is closed (620). Successiveiterations of the process of FIG. 6 are repeated until the end of thestream is reached (604). In various embodiments, the destinationdeduplication storage system is configured to perform subsequentprocessing to verify that all segments comprising all files included inthe reconstructed subset are present on the destination deduplicationstorage system; that all segments are stored only once on thedestination deduplication storage system; and/or to remove any segmentsthat may have been included in the data stream by virtue of beingincluded in the same container as a segment comprising a file includedin the reconstructed subset but which are not themselves associated withany file included in the reconstructed subset.

FIG. 7 is a flow diagram illustrating an embodiment of a process forreconstructing a subset of data using a copy of the subset as stored ona deduplication system. In the example shown, the reconstruction isperformed on a destination system other than deduplication storagesystem. In some embodiments, a software client, agent, applet, or othercode comprising computer instructions to implement the process of FIG. 7is installed and executed on the destination system. A next header inthe data stream is read (702). If the end of the stream has been reached(704), the process ends. Otherwise, if the header is a file header (706)a corresponding file is opened (708) on the destination system. Forexample, a file system running on the destination system is instructedto open a file having a name and directory location specified in thefile header. If the header instead identifies the current data in thestream as comprising container data (710), an offset indicating wherethe container data is located within the data stream is recorded (712).In some embodiments, the container offset is recorded in an offset mapor other data structure stored in memory in which container ID's andassociated offsets are recorded. In some embodiments, the container datais cached and/or indexed locally to generate index data usable to findsegments comprising the container data as cached. In some suchembodiments, container data offsets are not recorded. In someembodiments, the data stream includes data usable to determine how longto cache container data, for example a count of the number of timessegments within the container data are referenced in the stream. Oncethe container data as cached has been referenced the indicated number oftime, the container data is no longer held in the cache. If the headercomprises a segment reference (714), the corresponding segment data isobtained and written (716) to the file opened at (708). In someembodiments, the data stream is rewound to the location of the containerdata comprising the segment, as indicated by the previously storedoffset, the data is unpacked (e.g., decompressed), and the segment datacorresponding to the referenced segment is read (716). If the end of thecurrent file has been reached (718) the file is closed (720). Successiveiterations of the process of FIG. 7 are repeated until the end of thestream is reached (704).

FIG. 8 is a flow diagram illustrating an embodiment of a process forreconstructing a file from a data stream comprising a copy of a subsetof data as stored on a deduplication system. In various embodiments, 716of FIG. 7 comprises the process of FIG. 8. In the example shown, toobtain data comprising a referenced segment the container ID included inthe segment reference, and/or determined by referencing an index orother data identifying for each segment the corresponding container thecontainer data of which includes its corresponding segment data, is usedto read from the data stored at (712) of FIG. 7 a container data offsetindicating where the corresponding container data occurs in the datastream (802). The offset is used to rewind the data stream to the pointat which the container data occurs. The container data is read anddecompressed (804) and the segment data corresponding to the referencedsegment is written to the associated file (806). In some embodiments,the decompressed container data is (or at the option of an administratormay be) cached, for example to facilitate more rapid retrieval ofsegment data for other segments included in that container.

Using the approaches disclosed herein, a subset of data may be providedas a tape archive or other output from a deduplication storage system ina same de-duplicated (and in some embodiments compressed) form in whichit is stored on the deduplication storage system. By including containerdata for entire containers in the output stream the first time a segmentin the container is referenced, eliminating the need to decompress orotherwise unpack container data, and by using a bit mask or other easilyread and updated data structure to track which containers have alreadyhad their container data included in the stream, the desired outputstream can be generated quickly and efficiently.

Although the foregoing embodiments have been described in some detailfor purposes of clarity of understanding, the invention is not limitedto the details provided. There are many alternative ways of implementingthe invention. The disclosed embodiments are illustrative and notrestrictive.

What is claimed is:
 1. A method comprising: receiving an indicationidentifying a subset of data to be provided as output, wherein thesubset of data includes a first data segment; identifying a firstcontainer that includes the first data segment, wherein the firstcontainer may also include one or more other segment that are notincluded in the subset of data to be provided as output; determiningwhether corresponding container data associated with the first containerhas already been included in an output stream based at least in part ona data structure comprising for each of at least a subset of containersan indication of whether or not the corresponding container data hasbeen included in the output stream; in the event that it is determined,based at least in part on the data structure, that the first containercomprises container data that has already been included in the outputstream, not retrieving the corresponding container data associated withthe first container again and including the corresponding container datain the output stream; and in the event that it is determined, based atleast in part on the data structure, that the first container comprisescontainer data that has not already been included in the output stream:retrieving the corresponding container data associated with the firstcontainer and including the corresponding container data in the outputstream, wherein the corresponding container data includes the one ormore other segments that are not included in the subset of data to beprovided as output; and updating a corresponding value in the datastructure indicating that the container data associated with the firstcontainer has already been included in the output stream.
 2. The methodof claim 1, wherein the container data is included in the output streamin a form as it is stored in a de-duplicated storage system.
 3. Themethod of claim 2, wherein container data is stored in the de-duplicatedstorage system in a compressed and de-duplicated form.
 4. The method ofclaim 1, wherein the data structure comprises a bit mask.
 5. The methodof claim 4, wherein the bit mask comprises for each of said plurality ofcontainers a corresponding bit.
 6. The method of claim 1, furthercomprising writing the output stream to a tape or other removablestorage media.
 7. A data storage system, comprising: a processorconfigured to: receive an indication identifying a subset of data to beprovided as output wherein the subset of data includes a first datasegment; identify a first container that includes the first datasegments, wherein the first container may also include one or more othersegment that are not included in the subset of data to be provided asoutput; determine whether corresponding container data associated withthe first container has already been included in an output stream basedat least in part on a data structure comprising for each of at least asubset of containers an indication of whether or not the correspondingcontainer data has been included in the output stream; in the event thatit is determined, based at least in part on the data structure, that thefirst container comprises container data that has already been includedin the output stream, not retrieve the corresponding container dataassociated with the first container again and include the correspondingcontainer data in the output stream; and in the event that it isdetermined, based at least in part on the data structure, that the firstcontainer comprises container data that has not already been included inthe output stream: retrieve the corresponding container data associatedwith the first container and include the corresponding container data inthe output stream, wherein the corresponding container data includes theone or more other segments that are not included in the subset of datato be provided as output; and update a corresponding value in the datastructure indicating that the container data associated with the firstcontainer has already been included in the output stream; and a memorycoupled to the processor and configured to store the data structure. 8.The system of claim 7, wherein the container data is included in theoutput stream in a form as it is stored in a de-duplicated storagesystem.
 9. The system of claim 8, wherein container data is stored inthe de-duplicated storage system in a compressed and de-duplicated form.10. The system of claim 7, wherein the data structure comprises a bitmask.
 11. The system of claim 10, wherein the bit mask comprises foreach of said plurality of containers a corresponding bit.
 12. The systemof claim 7, wherein the processor is further configured to write theoutput stream to a tape or other removable storage media.
 13. A computerprogram product embodied in a non-transitory computer readable storagemedium and comprising computer instructions for: receiving an indicationidentifying a subset of data to be provided as output, wherein thesubset of data includes a first data segment; identifying a firstcontainer that includes the first data segment, wherein the firstcontainer may also include one or more other segment that are notincluded in the subset of data to be provided as output; determiningwhether corresponding container data associated with the first containerhas already been included in an output stream based at least in part ona data structure comprising for each of at least a subset of containersan indication of whether or not the corresponding container data hasbeen included in the output stream; in the event that it is determined,based at least in part on the data structure, that the first containercomprises container data that has already been included in the outputstream, not retrieving the corresponding container data associated withthe first container again and including the corresponding container datain the output stream; and in the event that it is determined, based atleast in part on the data structure, that the first container comprisescontainer data that has not already been included in the output stream:retrieving the corresponding container data associated with the firstcontainer and including the corresponding container data in the outputstream, wherein the corresponding container data includes the one ormore other segments that are not included in the subset of data to beprovided as output; and updating a corresponding value in the datastructure indicating that the container data associated with the firstcontainer has already been included in the output stream.
 14. Thecomputer program product of claim 13, wherein the container data isincluded in the output stream in a form as it is stored in ade-duplicated storage system.
 15. The computer program product of claim13, wherein the data structure comprises a bit mask.
 16. The computerprogram product of claim 15, wherein the bit mask comprises for each ofsaid plurality of containers a corresponding bit.
 17. The computerprogram product of claim 13, further comprising computer instructionsfor writing the output stream to a tape or other removable storagemedia.